Spark Architecture & RDD Basics
"Master Apache Spark's runtime cluster architecture — understanding Driver vs Executor JVMs, Client vs Cluster execution modes, and low-level Resilient Distributed Datasets (RDDs) with DAG lineage fault tolerance."
What You'll Master
Cluster Topology
Driver process coordination, Executor JVM worker pools, & Cluster Managers (YARN, K8s).
Execution Modes
Client Mode (edge-node interactive submission) vs Cluster Mode (production YARN execution).
RDD Lineage & DAGs
Immutability, lazy evaluation, & automatic partition recovery without data duplication.
Transformations & Actions
Narrow dependencies (map, filter) vs Wide dependencies (reduceByKey, groupByKey).
Learning Path & Course Syllabus
Driver vs Executor JVM processes, standalone/YARN/Kubernetes cluster managers, and Client vs Cluster deployment modes.
Low-level RDD core properties: immutability, partitioning, lazy evaluation, DAG graph lineage, and fault recovery.
Clickstream log dataset tracing through narrow transformations, wide shuffles, and driver action collections.
Scenario questions covering Driver OOM collect crashes, lineage fault tolerance, reduceByKey vs groupByKey shuffle boundaries, and Client vs Cluster modes.
What's Included in This Module
| Area | Overview |
|---|---|
| Topics Covered | Spark Driver & Executor JVMs, Cluster Managers, RDD Lineage, Narrow vs Wide Dependencies |
| Practical Code | PySpark RDD Transformations, Actions, & Interactive Jupyter Exercises |
| Assessments | 1 Hands-on RDD Log Processing Lab + 1 FAANG System Design Interview Quiz |